Text Summarization
Part 2, Chapter 15
In this chapter, we'll develop the text summarization piece of our application.
Newspaper3k
Our objective is to create a real-time text summarization service used for creating article summaries from a given URL.
We'll need to use a web scraping library to extract the text of an article and then use a machine learning library to generate the actual summary. We could roll our own solution with say Beautiful Soup (for scraping) and Bart (for summarizing), but there's a library that will do both: Newspaper3k.
So, start by adding the library to the requirements file:
newspaper3k==0.2.8
Install:
$ docker-compose up -d --build
Summarizer
Add a new file called project/app/summarizer.py:
# project/app/summarizer.py
from newspaper import Article
def generate_summary(url: str) -> str:
article = Article(url)
article.download()
article.parse()
article.nlp()
return article.summary
That's it.
After creating a new Article instance, we downloaded the given URL's HTML, extracted the meaningful content (via parse()) and the relevant keywords (via nlp()), and then generated a summary.
For more, review the Quickstart guide from the official docs.
The nlp method requires the punkt tokenizer from the Natural Language Toolkit (NLTK), so update summarizer.py like so to install it:
# project/app/summarizer.py
import nltk
from newspaper import Article
def generate_summary(url: str) -> str:
article = Article(url)
article.download()
article.parse()
try:
nltk.data.find("tokenizers/punkt")
except LookupError:
nltk.download("punkt")
finally:
article.nlp()
return article.summary
Next, update the post function in project/app/api/crud.py to generate the summary when a new URL is added:
async def post(payload: SummaryPayloadSchema) -> int:
article_summary = generate_summary(payload.url)
summary = TextSummary(url=payload.url, summary=article_summary)
await summary.save()
return summary.id
Don't forget the import:
from app.summarizer import generate_summary
Test it out:
$ http --json POST http://localhost:8004/summaries/ url=http://testdriven.io
HTTP/1.1 201 Created
content-length: 37
content-type: application/json
date: Mon, 27 Dec 2021 21:57:44 GMT
server: uvicorn
{
"id": 1,
"url": "http://testdriven.io"
}
$ http GET http://localhost:8004/summaries/1/
HTTP/1.1 200 OK
content-length: 779
content-type: application/json
date: Mon, 27 Dec 2021 21:58:23 GMT
server: uvicorn
{
"created_at": "2021-07-08T00:10:39.390969+00:00",
"id": 1,
"summary": "Our courses and tutorials teach practical application of popular tech used by both enterprise and startups.\nYou will quickly become effective in your chosen stack, as we provide pragmatic examples and detailed walkthroughs of the workings of each technology.\nThrough our courses, you will not only become more comfortable with specific tools and technologies like AWS ECS, Docker, and Flask, to name a few -- but you will also gain skills necessary to contribute to a team or launch your own exciting project.\nEvery tool taught in our courses is supported by large communities and is in high-demand by hiring managers around the globe.\nWe teach tools we love and use every day.",
"url": "http://testdriven.io"
}
Background Task
The generate_summary function introduces a blocking operation into the post function. Since both article.parse() and article.nlp() are expensive operations, let's move the entire generate_summary function out of the request/response flow by running it as a background task.
First, update post:
async def post(payload: SummaryPayloadSchema) -> int:
summary = TextSummary(url=payload.url, summary="")
await summary.save()
return summary.id
Next, update create_summary:
@router.post("/", response_model=SummaryResponseSchema, status_code=201)
async def create_summary(payload: SummaryPayloadSchema, background_tasks: BackgroundTasks) -> SummaryResponseSchema:
summary_id = await crud.post(payload)
background_tasks.add_task(generate_summary, summary_id, payload.url)
response_object = {"id": summary_id, "url": payload.url}
return response_object
Make sure to import both BackgroundTasks and generate_summary:
from fastapi import APIRouter, HTTPException, Path, BackgroundTasks
from app.summarizer import generate_summary
Finally, update generate_summary like so:
# project/app/summarizer.py
import nltk
from newspaper import Article
from app.models.tortoise import TextSummary
async def generate_summary(summary_id: int, url: str) -> None:
article = Article(url)
article.download()
article.parse()
try:
nltk.data.find("tokenizers/punkt")
except LookupError:
nltk.download("punkt")
finally:
article.nlp()
summary = article.summary
await TextSummary.filter(id=summary_id).update(summary=summary)
Now, the summary is generated after the response is sent back to the client. Once generated, the database is updated. To test, add asyncio.sleep to generate_summary:
# project/app/summarizer.py
import asyncio
import nltk
from newspaper import Article
from app.models.tortoise import TextSummary
async def generate_summary(summary_id: int, url: str) -> None:
article = Article(url)
article.download()
article.parse()
try:
nltk.data.find("tokenizers/punkt")
except LookupError:
nltk.download("punkt")
finally:
article.nlp()
summary = article.summary
await asyncio.sleep(10)
await TextSummary.filter(id=summary_id).update(summary=summary)
Then, add a new summary:
$ http --json POST http://localhost:8004/summaries/ url=http://testdriven.io
HTTP/1.1 201 Created
content-length: 34
content-type: application/json
date: Tue, 28 Dec 2021 00:09:22 GMT
server: uvicorn
{
"id": 5,
"url": "http://testdriven.io"
}
Get the summary:
$ http GET http://localhost:8004/summaries/5/
HTTP/1.1 200 OK
content-length: 134
content-type: application/json
date: Tue, 28 Dec 2021 00:09:39 GMT
server: uvicorn
{
"created_at": "2020-05-10T15:59:55.098074",
"id": 5,
"summary": "",
"url": "http://testdriven.io"
}
Wait about ten seconds before getting the summary again:
$ http GET http://localhost:8004/summaries/5/
HTTP/1.1 200 OK
content-length: 134
content-type: application/json
date: Tue, 28 Dec 2021 00:09:52 GMT
server: uvicorn
{
"created_at": "2020-05-10T15:59:55.098074",
"id": 5,
"summary": "Our courses and tutorials teach practical application of popular tech used by both enterprise and startups.\nYou will quickly become effective in your chosen stack, as we provide pragmatic examples and detailed walkthroughs of the workings of each technology.\nThrough our courses, you will not only become more comfortable with specific tools and technologies like AWS ECS, Docker, and Flask, to name a few -- but you will also gain skills necessary to contribute to a team or launch your own exciting project.\nEvery tool taught in our courses is supported by large communities and is in high-demand by hiring managers around the globe.\nWe teach tools we love and use every day.",
"url": "http://testdriven.io"
}
Be sure to remove asyncio.sleep once done.
Tests
Do the tests pass?
$ docker-compose exec web python -m pytest
You should see six failures:
> raise ArticleException('Article `download()` failed with %s on URL %s' %
(self.download_exception_msg, self.url))
E newspaper.article.ArticleException: Article `download()` failed with HTTPSConnectionPool(host='foo.bar', port=443): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f00e5582e50>: Failed to establish a new connection: [Errno -2] Name or service not known')) on URL https://foo.bar
Let's use monkeypatch again.
Example:
def test_create_summary(test_app_with_db, monkeypatch):
def mock_generate_summary(summary_id, url):
return None
monkeypatch.setattr(summaries, "generate_summary", mock_generate_summary)
response = test_app_with_db.post(
"/summaries/", data=json.dumps({"url": "https://foo.bar"})
)
assert response.status_code == 201
assert response.json()["url"] == "https://foo.bar"
Add the import:
from app.api import summaries
Fix the other failing tests on your own.
Run Flake8, Black, and isort:
$ docker-compose exec web flake8 .
$ docker-compose exec web black .
$ docker-compose exec web isort .
Commit your code and push it to GitHub. Test the summarizer on Heroku.
✓ Mark as Completed